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We search for the production of single top quarks via flavor-changing neutral current couplings 
of a gluon to the top quark and a charm (c) or up (u) quark. We analyze 230 pb -1 of lepton+jets 
data from pp collisions at a center of mass energy of 1.96 TeV collected by the DO detector at the 
Fermilab Tevatron Collider. We observe no significant deviation from standard model predictions, 
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and hence set upper limits on anomalous coupling parameters K c g /A and Kg /A, where the K g define 
the strength of the teg and tug couplings, and A defines the scale of new physics. The limits at 95% 
C.L. are: K c g /A < 0.15 TeV -1 and n u g / A < 0.037 TeV -1 . 

PACS numbers: 11.30.Hv; 13.85.Rm; 14.65.Ha; 14.70.Dj 



Top quarks were discovered in 1995 by the CDF and 
DO collaborations [l[ at the Fermilab Tevatron Collider 
in tt pair production involving strong interactions. The 
standard model (SM) also predicts the production of sin- 
gle top quarks via electroweak exchange of a W boson 
with cross sections of 0.88 pb in the s-channel (tb) and 
1.98 pb in the t-channel (tqb) 0. At the 95% C.L., limits 
set by DO are 6.4 pb on the s-channel cross section and 
5.0 pb on the i-channel cross section 0], and those set 
by CDF are 13.6 pb and 10.1 pb, respectively [jj. DO re- 
cently reported evidence for the production of single top 
quarks at significance of 3.4 standard deviations [5j. 

Since the top quark's discovery, several precision mea- 
surements have been made of its properties. Its large 
mass close to the electroweak symmetry-breaking scale 
suggests that any anomalous coupling is likely to be 
observed first in the top quark sector. One form of 
anomalous couplings can give rise to a single top quark 
in the final state through flavor-changing neutral cur- 
rent (FCNC) interactions with a charm or an up quark, 
involving the exchange of a photon, a Z boson, or a 
gluon [6] . Although such interactions can be produced by 
higher-order radiative corrections in the SM, the effect is 
too small to be observed ■ Any observable signal indi- 
cating the presence of such couplings would be evidence 
of physics beyond the SM and would shed additional light 
on flavor physics in the top quark sector. 

At present, strong constraints exist for FCNC pro- 
cesses via a photon or a Z boson exchange [1, 0, [l(| 
from studies of both the production and decay of top 
quarks. In this Letter, we present a search for produc- 
tion of single top quarks via FCNC couplings of a gluon 
to the top quark in data collected from pp collisions at 
y/s = 1.96 TeV using the DO detector. This is the first 
search of its kind at hadron colliders. We consider top 
quark production rather than decay, since the former is 
more sensitive to the anomalous couplings (n g ) involv- 
ing the gluon ;11|. To date, the best constraints on 
these processes are from the DESY ep Collider (HERA): 
K g / A < 0.4 TeV" 1 , at 95% C.L. [12], where A is the new 
physics cut-off scale. 

In this analysis, we consider events where the top 
quark decays into a b quark and a W boson, and the 
latter subsequently decays leptonically (W — » £v, where 
I = e, ji or t, with the r decaying to either an elec- 
tron or a muon, and two neutrinos). This gives rise to 
an event with a charged lepton of high transverse mo- 
mentum (pr), significant missing transverse energy 
from the neutrinos, and at least two jets, one that is a b- 
quark jet (from the top quark decay), and the other from 



a c quark, u quark, or a gluon. Displaced secondary ver- 
tices are used to identify b jets @. The largest physics 
backgrounds to these events are from SM production of 
VF+jets and it, along with smaller contributions from SM 
production of single top quarks (tb and tqb) and dibosons 
(WW and W Z). An additional source of background is 
from multijet events in which a jet is incorrectly iden- 
tified as an electron or in which a muon from a heavy 
flavor decay appears isolated. 

The DO detector is described elsewhere [13| |. We use 
the same dataset, basic event selections and background 
modeling as in our SM single top quark search [3J]; how- 
ever, since the FCNC signal processes have only one b 
quark in the final state, we consider here events with 
only one ^-tagged jet. In addition, we include here the 
SM single top quark processes (tb and tqb) in the back- 
ground model. The data were recorded between August 
2002 and March 2004 with a total integrated luminosity 
of 230 ±15 pb -1 [3] and were collected using a trigger 
that required a reconstructed jet and an electromagnetic 
energy cluster in the electron channel, or a jet and a 
muon in the muon channel. 

We model the FCNC signal kinematics using a parton- 
level leading order (LO) matrix element event generator 
CompHEP We consider the following four sub- 

processes: 

cq ->tq , cg^tg, qq -> tc, gg -> tc, (1) 

and also those that replace the c quark with a u quark and 
the charge conjugates. The identity of the associated fi- 
nal state jet depends upon the initial state of the system. 
Decays of the top quark and W boson are done in Com- 
pHEP to take into account all spin-dependent effects. 
The effects of FCNC couplings are parameterized in a 
model-independent way via an effective Lagrangian (llT | 
that is a linear function of the factor n g /A. The pro- 
duction cross section of single top quarks thus depends 
quadratically on K g /A, and for certain values of K g / A can 
be significantly larger than that in the SM, as shown in 
Table |TJ The cross sections are evaluated at a top quark 
mass of m t — 175 GeV, with the factorization and renor- 
malization scales set to Q 2 = mf . The LO cross sections 
are scaled to next-to-leading (NLO) order by a if-factor 
(NLO/LO cross section ratio) of 1.6 (l7| . 

The effect of FCNC couplings on the top quark decay 
is negligible for n g /A <, 0.2 TeV -1 [ll|. In this range of 
K g / A, it is therefore safe to assume that the top quark 
decays into a W boson and a b quark with a branching 
fraction close to unity, as in the standard model, and 
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TABLE I: The production cross sections o(t) of single 
top quarks through a gluon exchange in pp collisions at 
y/s = 1.96 TeV for different k 9 /A values, as obtained from 
CompHEP and scaled to NLO by a if -factor of 1.6. 



TABLE II: Summary of event selections. 



k 9 /A [TeV -1 ] 


a 


(t) [pb] 




teg 


tug 




K = 


0) = 0) 


0.01 


0.05 


0.88 


0.03 


0.45 


7.92 


0.07 


2.40 


42.61 


0.11 


5.86 


104.78 





Electron channel Muon channel 


Lepton 


E T > 15 GeV p T > 15 GeV 

\n\ < i.i H < 2.0 




15 < E T < 200 GeV 


Jets 


2, 3 or 4 jets, E T > 15 GeV, \r)\ < 3.4 
£ T (jetl) > 25 GeV, |r?(jetl)| < 2.5 


b jets 


exactly one tagged jet 



hence, the cross section a(t) multiplied by the branch- 
ing fraction for the process t — » Wb — ► tvb would also 
depend quadratically on K g /A. We may therefore model 
the shapes of the signal kinematic variables at any one 
value of Kg/ A and scale the distributions appropriately 
to obtain them at any other value of the coupling. We 
choose that value of n g /A to be 0.03 TeV -1 in Com- 
pHEP and generate two sets of signal events: one for 
the teg process only, in which Kg is set to zero, and the 
other for the tug process only, in which K g is set to zero. 

The parton-level s amp les from CompHEP are pro- 
cessed with PYTHIA |l8| for fragmentation, hadroniza- 
tion, and modeling of the underlying event, using the 
CTEQ5l 11911 parton distribution functions. We use 
TAUOLA [20] for the tau lepton decays and evtgen 21 1 
for the 6-hadron decays. The generated events are pro- 
cessed through a GEANT-based 22] simulation of the 
DO detector, and normalized to the NLO cross sections 
for Kg/ A — 0.03 TeV -1 . For the backgrounds, the Monte 
Carlo (MC) simulated samples are generated and nor- 
malized as described in Ref. Q. 

The event selections 0] applied to the simulated signals 
and backgrounds and to the DO data are summarized in 
Table [TTJ The resulting numbers of events from all sam- 
ples, along with their systematic uncertainties described 
later, are shown in Table [TTtI We find that the observed 
numbers of events agree with the predicted numbers for 
the SM backgrounds within uncertainties in both the 
electron and muon channels, and that the FCNC signals 
are a tiny fraction. We therefore construct multivari- 
ate discriminants using neural networks to separate the 
expected signal from the background and enhance the 
sensitivity. 

We use MLPfit implementation [23| of neural networks 
with ten input variables, one hidden layer, and one out- 
put layer. The input variables represent individual object 
kinematics, global event kinematics, and angular corre- 
lations, and are listed in Table IIVI For training of the 
networks, we consider the sum of teg and tug processes 
as signal, since the final states for these two processes 
are indistinguishable in this analysis, and the sum of 
all SM processes as background. The processes in each 



TABLE III: Estimates of the FCNC signal and background 
yields, and the number of observed events in data after all 
selections, for the electron and muon channels. The signal 
yields are evaluated at k 9 /A = 0.03 TeV -1 . The yields for ti 
include both lepton+jets and dilepton final states, and those 
from W+jets also include the diboson backgrounds. 



Source 


Electron channel Muon channel 


teg 


0.6 ±0.2 


0.6 ±0.2 


tug 


8.4 ±2.1 


9.8 ±2.7 


SM single top (tb+tgb) 


6.4 ±1.4 


6.1 ±1.4 


ti 


31.8 ±6.9 


31.4 ±7.0 


W+jets 


84.6 ±10.2 


76.8 ±8.5 


Multijets 


13.7 ±4.3 


17.2 ±1.5 


Total SM background 


136.5 ±13.4 


131.5 ±12.7 


Observed no. of events 


134 


118 



sum are weighted according to their relative proportions 
in the 230 pb -1 of data. The resulting networks sepa- 
rate the FCNC signals not only from the dominant back- 
grounds (W±jets and ti) but also from the SM single top 
quark processes. Output distributions for the combined 
electron and muon channels are shown in Fig. [1] for the 
summed background samples and DO data. Also shown is 
the FCNC signal with the teg and tug processes evaluated 
at K g / A — 0.03 TeV -1 and summed. Since the observed 
spectrum agrees well with the predicted SM background, 
we set upper limits on the FCNC coupling parameters 
Kg / A and Kg/A. 

To estimate systematic uncertainties, we consider two 
classes of effects: those that alter the overall normaliza- 
tion of the distributions and those that also change their 
shapes. The dominant normalization effects are from lep- 
ton identification (4%), integrated luminosity measure- 
ment (6.5%), and cross section estimates. The uncertain- 
ties on the cross sections vary from 9% for diboson pro- 
duction to 16% for SM sing le top quark production and 
18% for the tt samples [26|. The latter two include the 
uncertainty due to the top quark mass. For the FCNC 
signal, we factor out the parameter (k 9 /A) 2 from the 
cross section, and assume an uncertainty of 15% on the 
remaining quantity based on a discussion in Ref. [13] on 
how the theoretical predictions depend on the particular 
choice of the factorization scale. The 1 / F±jets and multi- 
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TABLE IV: Input variables used in the neural network analysis. 



PT (jctl) 
PT(jetl taggcd ) 
77(lepton) 
#T 

p T (jetl,jet2) 
ff T (jetl,jet2) 
Pt(W) 

M(W,jetl taggod ) 

A/(alljets) 

cos(jetl,lepton)i ab 



Transverse momentum of the leading jet 
Transverse momentum of the 6-tagged jet 
Pseudorapidity [24] of the lepton 
Missing transverse energy 

Transverse momentum of the two leading jets 

Scalar sum of the transverse momenta of the two leading jets 

Transverse momentum of the reconstructed W boson 

Invariant mass of the reconstructed top quark using the W boson [25] and the 6-tagged jet 
Invariant mass of all jets 

Cosine of the angle between the leading jet and lepton in the laboratory frame of reference 



♦D0 230pb 1 
-FCNC(x10) 
SM tb+tqb 




0.5 1 

NN output 

FIG. 1: Neural network output distributions of summed 
background samples and DO data, for the combined elec- 
tron and muon channels. Also shown is the FCNC sig- 
nal distribution with the teg and tug processes evaluated at 
K g /A = 0.03 TeV -1 and summed (color online). 



jets samples have an overall uncertainty of 4% from their 
normalization to data Q . This includes an uncertainty of 
25% on the heavy flavor fraction of the VF+jets sample. 



The shape effects are modeled by shifting the uncer- 
tainty components one-by-one by plus or minus one stan- 
dard deviation with respect to their nominal values, for 
each sample, and propagating the changes to the kine- 
matics of the different objects (electrons, muons, jets, 
and $t) before making any event selections. The result- 
ing uncertainties are as follows: (i) (1-16)% due to jet 
energy scale, (ii) (2-8)% from trigger modeling, (iii) (1- 
5)% due to jet energy resolution, (iv) (1-9)% due to jet 
identification, and (v) (5-13)% from 6-tag modeling. Al- 
though the W+jets MC yield is normalized to data, it is 
also affected by the uncertainty from the 6-tag modeling 
since the normalization is done before &-tag parametriza- 
tion, and we take this into account. 

We use a Bayesian approach to set upper limits [27| 
on the FCNC coupling parameters. Given N observed 
events, we define the Bayesian posterior probability den- 
sity in a two-dimensional plane of (Kg/A) 2 and (Kg/A) 2 
as: 



MK/A] 2 ,K/A] 2 | AO cx 



L(N | n) Pl (f c J u ,b) p 2 (K/A} 2 ) P 3 (K/A} 2 ) d/ c d/„d&, 



(2) 



r 



where L is a Poisson likelihood with mean n, and pi (i = 
1,2,3) are the prior probability densities of the respec- 
tive parameters. The likelihood L is a product of the 
likelihoods over all bins of the neural network output 
distributions, n is the predicted number of events, equal 
to the sum of signal (s) and background (b) yields: 



(3) 



= f B x (</A) 2 + f u x (k u JA) 2 + b, 



where the constants f c and f u are determined from the 
simulated signal samples at K g /A = 0.03 TcV -1 . The 
prior probability density, pi, is a multivariate Gaussian, 
with the mean and standard deviation defined by the 
estimated yields and their uncertainties, to take into ac- 
count correlations among the different samples and bins. 
Since the signal cross sections depend quadratically on 



Kg/ A, for p2 and ps we choose priors fiat in (k°/A) 2 and 
(k^ /A) 2 respectively, which imply priors flat in the cor- 
responding cross sections. 

From the two-dimensional posterior probability den- 
sity, exclusion contours at different levels of confidence 
(k) are defined as contours of equal probability that en- 
close a volume k around the peak of the posterior prob- 
ability density. These contours are shown in Fig. [2j 
using data from both the electron and muon channels 
and including all systematic uncertainties with correla- 
tions. The one-dimensional posterior probability density 
over any dimension is obtained by integrating the two- 
dimensional posterior over the other dimension. The re- 
sulting limits, translated to K g /A, using data (observed 
limits) as well as the expected limits for which the ob- 
served count is set to the predicted background yield in 
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any bin, are summarized in Table [V] 

To conclude, we analyzed 230 pb" 1 of lepton+jets data 
collected at DO from pp collisions at a center of mass en- 
ergy of 1.96 TeV, and searched for the presence of non- 
SM production of single top quarks. We found no de- 
viation from SM predictions, and therefore set limits on 
the anomalous coupling parameters, K c g / A and Kg/ A, us- 
ing multivariate neural network discriminants. The 95% 
C.L. observed (expected) limits are 0.15 (0.16) TeV" 1 on 
K c g /A, and 0.037 (0.041) TeV" 1 on k u JA. These are the 
first limits from hadron colliders on FCNC couplings of 
a gluon to the top quark and a charm or up quark, and 
a factor 3-11 better than those from HERA. 



% 0.003 




0.02 0.03 0.04 
(k=/A) 2 [TeV 2 ] 

FIG. 2: Exclusion contours at various levels of confidence 
using 230 pb -1 of DO data in both the electron and muon 
channels (color online). 



TABLE V: Upper limits on k% /A and Kg/A, at 95% C.L. 





Observed (expected) limits [TeV 1 ] 

k c 3 /A </A 


Electron channel 
Muon channel 
Combined 


0.16 (0.19) 0.046 (0.052) 
0.21 (0.21) 0.049 (0.050) 
0.15 (0.16) 0.037 (0.041) 
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